(9) CLAIMS 



1 . A system for accessing a substantially comprehensive record of an 
immediate environment of a user, comprising: 

a substantially continuous record of information from a plurality of 

time-correlated input data streams; 

means for specifying a query into the record; and 

means for displaying a result of a query made using said means for 

specifying a query. 

2. The system of claim 1 comprising: 

the input data streams are taken from a set consisting of video data, audio 
data, data representative of the user's location and changes thereof, data 
representative of measurements of the user's physical state, and data 
representative of local ambient environment conditions of said immediate 
environment, and current and reference standard time. 

3. The system of claim 1 , further comprising: 
means for collecting said record. 

4. The system of claim 1 comprising: 

the plurality of time-correlated input data streams includes at least one 
annotation data stream associated with at least one of said time-correlated input 
streams. 
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5. The system of claim 4 comprising: 

the annotation data stream is from a set consisting of transcription of 
speech, information representative of people present on camera or speaking, 
information representative of location, information representative of user activities, 
information representative of aspects of video and audio input not directly input by 
said collecting means, information representative of the user's biometric or 
emotional states, and information representative of other events of said immediate 
environment not directly input by said collecting means. 

6. The system of claim 4, further comprising: 
means for providing said annotation data stream. 

7. The system of claim 6 comprising: 

the means for providing said annotation data stream is an input device by 
which said user generates annotations. 

8. The system of claim 6 comprising: 

the means for providing said annotation data stream is an automatic input 
device wherein annotation is implemented based on other data in the record. 

9. The system of claim 8 wherein the means of providing said annotation data 
stream is triggered by detection that data in at least one of said time-correlated 
input data streams matches a predetermined pattern. 
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10. The system of claim 9, further comprising: 

means for specifying at least one said predetermined pattern and an 
annotation representative thereof to insert in the record when the predetermined 
pattern is detected. 

1 1 . The system of claim 6 comprising: 

the means for providing said annotation data stream includes means for 
using a remote data source. 

12. The system of claim 1 comprising: 

the means for displaying a result includes means for presenting a 
time-bounded segment of at least one of said time-correlated input streams from 
the record. 

13. The system of claim 12 comprising: 

said means for presenting provides a presentation of a time-bounded 
segment of a plurality of said streams from the record wherein the presentations are 
overlaid. 

14. The system of claim 13, wherein one of said streams is a video data stream, 
comprising: 

overlaying the video data stream with subtitles visible during video playback. 
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15. The system of claim 13, wherein one of said streams is a video data stream, 
comprising: 

overlaying the video data stream by inserting word balloons visible during 
video playback. 

16. The system of claim 4 comprising: 

the means for displaying a result of a query includes means for presenting a 
set of annotations. 

1 7. The system of claim 4 comprising: 

the means for specifying a query includes means for identifying a target 
time-bounded segment of the record. 

18. The system of claim 17 comprising: 

the means for identifying includes means for finding a segment based on the 
attributes contained in the record during the segment and means for specifying a 
set of attributes of interest for testing whether the segment matches. 

1 9. The system of claim 1 8, further comprising: 

said set of attributes includes a similarity metric, wherein the means for 
identifying identifies the target time-bounded segment based on strength of 
similarity under the similarity metric to a second segment; and 

means of identifying the second segment. 
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20. The system of claim 19 wherein the second segment is an interval of time 
immediately prior to the present. 

21 . The system of claim 1 , further comprising: 

means for identifying a plurality of users and for proving system use 
authorization for the users . 

22. The system of claim 21 comprising: 

the means for displaying requires proof of authorization via said means for 
identifying a plurality of users and for proving system use authorization before 
displaying data from the record. 

23. The system of claim 21 , further comprising: 

means for identifying a subset of the record and for providing authorization 
for viewing the subset to another person or system. 

24. The system of claim 1 , further comprising: 

means of extracting a subset of the record and transmitting an extracted 
said subset to a secondary system or a device compatible with the system. 

25. The system of claim 1 , further comprising: 

means for compressing the record by deleting information. 
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26. The system of claim 25 comprising: 

the means for compressing the record includes a level-of-user-interest 
metric, deleting information based upon identifying uninteresting information in the 
record in accordance with said level-of-user-interest metric. 

27. The system of claim 26 wherein the interestingness metric consists of one or 
more criteria including changes in annotation, biometrically-determined interest or 
excitement level, frame-to-frame video change, overall similarity between a 
selected segment of said time-correlated input data streams and segments which 
precede said selected segment. 

28. The system of claim 26 wherein the deleting information is based upon 
reducing the frame rate of a video data stream of said time-correlated input data 
streams . 

29. The system of claim 26 wherein the plurality of time-correlated input data 
streams includes at least one annotation data stream associated with at least one 
specified one of said time-correlated input streams and wherein the deleting of 
information is based upon discarding input stream data and retaining annotations 
of said at least one annotation data stream. 

30. The system of claim 26, wherein the record comprises a portable part and 
an archival part, the system further comprising: 

means for transmitting a subset of the record wherein deleting information 
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is based upon transmitting information to be deleted from the portable part to the 
archival part. 

31 . A portable device for capturing a substantially comprehensive record of an 
immediate environment of a user, comprising: 

a portable housing; 

associated with said housing, one or more data collection devices; and 
integrated with said housing, a time-keeping device, a data storage device, 
and a programmable device for correlating all data captured by the data collection 
devices based upon time reported by the time-keeping device and for storing so- 
correlated data on the storage device. 

32. The device of claim 31 , further comprising: 

associated with the housing at least one input device, the programmable 
device further operable to use input from each said input device to record 
time-stamped annotations in the record. 

33. The device of claim 31 comprising: 

the programmable device having program code for determining a segment 
of the record to be uninteresting and to compress the record by deleting or 
degrading segments so-determined. 

34. The device of claim 31 , further comprising: 

a communications port for transmitting a subset of the record stored on the 
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data storage device. 

35. A wearable device for querying a substantially comprehensive record of an 
immediate environment of a user, comprising: 

a portable housing; 

associated with said housing, at least one data output device, and at least 
one input device for specifying data queries; and 

integrated with said housing, a data storage device containing a data record 
and a programmable device for accepting queries from the at least one input 
device, for identifying sections of the record based on the queries, and for 
displaying the results of the queries on the at least one output device. 

36. The device of claim 35, further comprising: 

associated with the housing, an identification device for identifying a user as 
authorized to make said queries. 

37. A system for data collection comprising: 
a portable housing; and 

interconnected within said housing, data collection means for capturing data 
representative of an immediate vicinity of the system, integration means for 
correlating said autobiographical data, and memory means for storing said 
autobiographical data. 
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38. The system as set forth in claim 37, said data collection means comprising: 
audio-video devices for capturing photo-images and accompanying sound in 

the immediate vicinity of said system. 

39. The system as set forth in claim 38, said audio-video devices further 
comprising: 

a camera data processor, and 

a remote camera lens for transmitting photo-images to said camera data 
processor. 

40. The system as set forth in claim 38, said audio-video devices further 
comprising: 

an audio signal processor, and 

a remote microphone for transmitting audio signals to said audio signal 
processor. 

41 . The system as set forth in claim 37, said data collection means comprising: 
global positioning system (GPS) for capturing GPS data of said system. 

42. The system as set forth in claim 37, said data collection means comprising: 
connected to an end-user of the system, biometric sensors for capturing 

system user biometric readings. 



-33- 



Docket No.: 1001 0971-1 



43. The system as set forth in claim 37, said data collection means comprising: 
environmental sensors for capturing environmental condition data in the 

immediate vicinity of said system. 

44. The system as set forth in claim 37, said data collection means further 
comprising: 

connected to said integration means, at least one direct data input port for 
receiving digital data from a remote source. 

45. The system as set forth in claim 37, further comprising: 
feedback means for controlling said system and for monitoring data 

collection in real time. 

46. The system as set forth in claim 45, said feedback means further 
comprising: 

a heads-up display for monitoring data which comprises perceivable 
stimulus data. 

47. The system as set forth in claim 45, said feedback means further 
comprising: 

an electromechanical display and command entry panel. 

48. The system as set forth in claim 45, said feedback means further 
comprising: 
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an audio transducer for audibly transmitting sound data collected by said 
data collection means. 

49. The system as set forth in claim 37, said integration means further 
comprising: 

a programmable device including routines for real-time integrating of the 
autobiographical data, storing the autobiographical data, and downloading 
autobiographical data to an off-board data storage apparatus. 

50. The system as set forth in claim 45, said feedback means further 
comprising: 

means for retrieving autobiographical data selectively based on 
predetermined search parameters. 

51 . The system as set forth in claim 49, further comprising: 
routines for editing 323, 325 the autobiographical data. 

52. A process for generating autobiographical data, the method comprising: 
providing an integrated apparatus for collecting data representative of 

perceptual stimuli in the immediate vicinity of a person; 
continuously collecting said data; 

in real-time, integrating said data in accordance with predetermined 
relational characteristics of said perceptual stimuli into a content retrievable data 
collection; and 
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storing said data collection in a memory. 

53. The process as set forth in claim 52, said collecting further comprising: 
receiving time-correlated input data streams representative of real-time 

information related to visual stimuli, audio stimuli, geophysical position, local 
environmental conditions, and autobiographical biophysical condition of said 
person. 

54. The process as set forth in claim 53 wherein said data streams include 
annotation data. 

55. The process as set forth in claim 54 wherein said annotation data is 
selected from a group including transcription of speech data, a digital marker 
related to at least one of said data streams, a digital marker related to an interest 
level of at least one of said streams, a digital marker related to biometric condition 
of said person. 

56. The process as set forth in claim 54 wherein said annotation data is 
manually provided by said person. 

57. The process as set forth in claim 54 wherein said annotation data is 
automatically generated based on other data in a current one of said data streams. 
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58. The process as set forth in claim 57 wherein said annotation data is 
automatically generated based upon detection that said current one of said data 
streams matches a pattern of data in said memory. 

59. The process as set forth in claim 54 wherein said annotation data is 
provided from a remote data source. 

60. The process as set forth in claim 54 further comprising: 
displaying data from said collection. 

61 . The process as set forth in claim 60, said displaying further comprising: 
presenting a time-bounded segment of at least one stream of said data. 

62. The process as set forth in claim 61 said presenting further comprising: 
overlaying a plurality of streams of said data. 

63. The process as set forth in claim 62 wherein said annotation data for 
overlaying of a video data stream and a transcription data stream is by inserting 
subtitles in the video data stream. 

64. The process as set forth in claim 62 wherein annotation data for overlaying 
of a video data stream and a transcription data stream is by inserting a word 
balloon in the video data stream. 
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65. The process as set forth in claim 60, said displaying further comprising: 
presenting a set of said annotations as a response to specifying a query. 

66. The process as set forth in claim 65, said specifying further comprising: 
identifying a target time-bounded segment of said collection. 

67. The process as set forth in claim 66, said identifying further comprising: 
specifying a set of attributes, and 

finding the target time-bounded segment based on said attributes contained 
in the collection during the target time-bounded segment. 

68. The process as set forth in claim 66, said identifying further comprising: 
specifying a similarity metric related to a first time-bounded segment, and 
finding the target time-bounded segment based on a said similarity metric. 

69. The process as set forth in claim 68 wherein said first time-bounded 
segment is immediately prior to present time. 

70. The process as set forth in claim 54, further comprising: 
on-demand, compressing said data collection. 

71 . The process as set forth in claim 70, said compressing further comprising: 
specifying deletion parameters, and 

deleting information based upon said parameters. 
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72. The process as set forth in claim 71 wherein said deletion parameters 
include a parameter representative of data level of interest of the person. 

73. The process as set forth in claim 72 wherein parameter representative of 
data level of interest of the person is selected from a group comprising changes in 
annotation, biometrically-determined interest or excitement level, frame-to-frame 
video change, and overall similarity between a current segment and earlier time- 
bounded segments. 

73. The process as set forth in claim 71 , said deleting further comprising: 
deleting specified time-bounded segments while retaining the annotations 

related thereto. 

74. The process as set forth in claim 52 wherein said memory includes a 
portable memory associated with said collecting and an archival memory and said 
storing further comprises transmitting a subset of said collection from said portable 
memory to said archival memory. 

75. Surveillance apparatus comprising: 
a portable housing; and 

integrated with said housing, a camera, an audio recorder, a GPS, a data 
memory, and a programmable device for integrating all data captured by said 
camera, audio recorder, and GPS into an integrated, content-retrievable format 
and for storing and retrieving data so formatted from said memory. 
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76. The apparatus as set forth in claim 75, further comprising: 

associated with said housing a biometric monitor for connecting to a subject, 

and 

said programmable device including routines for integrating data collected 
by said monitor with said all data captured by said camera, audio recorder and 
GPS. 

77. The apparatus as set forth in claim 75, further comprising: 

associated with said housing a local environmental condition data recorder, 

and, 

said programmable device including routines for integrating data collected 
by said monitor with said all data captured by said camera, audio recorder and 
GPS. 

78. The system as set forth in claim 75, said programmable device further 
comprising: 

program code for analyzing a segment of said data based on at least one 
predetermined parameter and for compressing the segment based on said 
analyzing. 

79. The system as set forth in claim 75, further comprising: 

a mechanism for transmitting said captured data from said system. 
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80. The system as set forth in claim 75, further comprising: 

associated with said housing, a mechanism for displaying said captured 

data. 

81 . The system as set forth in claim 80, said programmable device further 
comprising: 

program code for accepting a data query, for identifying segments of the 
captured data based on said query, and for displaying said segments on said 
mechanism for displaying. 
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